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Preface  &  Acknowledgements 


Welcome  to  our  Ninth  Annual  Acquisition  Research  Symposium!  This  event  is  the 
highlight  of  the  year  for  the  Acquisition  Research  Program  (ARP)  here  at  the  Naval 
Postgraduate  School  (NPS)  because  it  showcases  the  findings  of  recently  completed 
research  projects — and  that  research  activity  has  been  prolific!  Since  the  ARP’s  founding  in 
2003,  over  800  original  research  reports  have  been  added  to  the  acquisition  body  of 
knowledge.  We  continue  to  add  to  that  library,  located  online  at 

www.acquisitionresearch.net,  at  a  rate  of  roughly  140  reports  per  year.  This  activity  has 
engaged  researchers  at  over  60  universities  and  other  institutions,  greatly  enhancing  the 
diversity  of  thought  brought  to  bear  on  the  business  activities  of  the  DoD. 

We  generate  this  level  of  activity  in  three  ways.  First,  we  solicit  research  topics  from 
academia  and  other  institutions  through  an  annual  Broad  Agency  Announcement, 
sponsored  by  the  USD(AT&L).  Second,  we  issue  an  annual  internal  call  for  proposals  to 
seek  NPS  faculty  research  supporting  the  interests  of  our  program  sponsors.  Finally,  we 
serve  as  a  “broker”  to  market  specific  research  topics  identified  by  our  sponsors  to  NPS 
graduate  students.  This  three-pronged  approach  provides  for  a  rich  and  broad  diversity  of 
scholarly  rigor  mixed  with  a  good  blend  of  practitioner  experience  in  the  field  of  acquisition. 
We  are  grateful  to  those  of  you  who  have  contributed  to  our  research  program  in  the  past 
and  hope  this  symposium  will  spark  even  more  participation. 

We  encourage  you  to  be  active  participants  at  the  symposium.  Indeed,  active 
participation  has  been  the  hallmark  of  previous  symposia.  We  purposely  limit  attendance  to 
350  people  to  encourage  just  that.  In  addition,  this  forum  is  unique  in  its  effort  to  bring 
scholars  and  practitioners  together  around  acquisition  research  that  is  both  relevant  in 
application  and  rigorous  in  method.  Seldom  will  you  get  the  opportunity  to  interact  with  so 
many  top  DoD  acquisition  officials  and  acquisition  researchers.  We  encourage  dialogue  both 
in  the  formal  panel  sessions  and  in  the  many  opportunities  we  make  available  at  meals, 
breaks,  and  the  day-ending  socials.  Many  of  our  researchers  use  these  occasions  to 
establish  new  teaming  arrangements  for  future  research  work.  In  the  words  of  one  senior 
government  official,  “I  would  not  miss  this  symposium  for  the  world  as  it  is  the  best  forum 
I’ve  found  for  catching  up  on  acquisition  issues  and  learning  from  the  great  presenters.” 

We  expect  affordability  to  be  a  major  focus  at  this  year’s  event.  It  is  a  central  tenet  of 
the  DoD’s  Better  Buying  Power  initiatives,  and  budget  projections  indicate  it  will  continue  to 
be  important  as  the  nation  works  its  way  out  of  the  recession.  This  suggests  that  research 
with  a  focus  on  affordability  will  be  of  great  interest  to  the  DoD  leadership  in  the  year  to 
come.  Whether  you’re  a  practitioner  or  scholar,  we  invite  you  to  participate  in  that  research. 

We  gratefully  acknowledge  the  ongoing  support  and  leadership  of  our  sponsors, 
whose  foresight  and  vision  have  assured  the  continuing  success  of  the  ARP: 

•  Office  of  the  Under  Secretary  of  Defense  (Acquisition,  Technology,  & 
Logistics) 

•  Director,  Acquisition  Career  Management,  ASN  (RD&A) 

•  Program  Executive  Officer,  SHIPS 

•  Commander,  Naval  Sea  Systems  Command 

•  Program  Executive  Officer,  Integrated  Warfare  Systems 

•  Army  Contracting  Command,  U.S.  Army  Materiel  Command 
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•  Office  of  the  Assistant  Secretary  of  the  Army  (Acquisition,  Logistics,  & 
Technology) 

•  Deputy  Director,  Acquisition  Career  Management,  U.S.  Army 

•  Office  of  Procurement  and  Assistance  Management  Headquarters, 
Department  of  Energy 

•  Director,  Defense  Security  Cooperation  Agency 

•  Deputy  Assistant  Secretary  of  the  Navy,  Research,  Development,  Test  & 
Evaluation 

•  Program  Executive  Officer,  Tactical  Aircraft 

•  Director,  Office  of  Small  Business  Programs,  Department  of  the  Navy 

•  Director,  Office  of  Acquisition  Resources  and  Analysis  (ARA) 

•  Deputy  Assistant  Secretary  of  the  Navy,  Acquisition  &  Procurement 
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Reuben  Pitts — Mr.  Pitts  is  the  president  of  Lyceum  Consulting.  He  joined  the  Naval  Weapons  Lab  in 
Dahlgren,  VA,  in  June  1968  after  graduating  from  Mississippi  State  University  with  a  BSME.  His  early 
career  was  spent  in  ordnance  design  and  weapons  systems.  He  subsequently  served  on  the  planning 
team  to  reintroduce  the  Navy  to  Wallops  Island,  VA,  currently  a  multiple  ship  combat,  over-the-water 
weapons  testing  lab  for  Surface  Ship  Combat  Systems,  Fighter  Aircraft,  and  live  missile  firings.  His 
outstanding  service  as  the  deployed  Science  Advisor  to  Commander,  U.S.  Sixth  Fleet  was 
recognized  with  the  Navy’s  Superior  Civilian  Service  (NSCS)  Award  and  the  Navy  Science 
Assistance  Program  Science  Advisor  of  the  Year  Award. 

Mr.  Pitts  was  selected  to  lead  the  technical  analysis  team  in  support  of  the  formal  JAG 
investigation  of  the  downing  of  Iran  Air  Flight  655  by  USS  Vincennes ,  and  participated  in  subsequent 
briefings  to  CENTCOM,  the  Chairman  of  the  Joint  Chiefs,  and  the  Secretary  of  Defense.  As  head, 
Surface  Ship  Program  Office  and  Aegis  program  manager,  Mr.  Pitts  was  awarded  a  second  NSCS, 
the  James  Colvard  Award,  and  the  John  Adolphus  Dahlgren  Award  (Dahlgren’s  highest  honor)  for  his 
achievements  in  the  fields  of  science,  engineering,  and  management.  Anticipating  the  future  course 
of  combatant  surface  ships,  Mr.  Pitts  co-founded  the  NSWCDD  Advanced  Computing  Technology 
effort,  which  eventually  became  the  Aegis/DARPA-sponsored  High  Performance  Distributed 
Computing  Program;  the  world’s  most  advanced  distributed  real-time  computing  technology  effort. 
That  effort  was  the  foundation  for  the  Navy’s  current  Open  Architecture  Initiative. 

In  2003  Mr.  Pitts  accepted  responsibility  as  technical  director  for  PEO  Integrated  Warfare 
Systems  (IWS),  the  overall  technical  authority  for  the  PEO.  In  September  of  that  year,  he  was 
reassigned  as  the  major  program  manager  for  Integrated  Combat  Systems  in  the  PEO.  In  this 
position,  he  was  the  program  manager  for  the  Combat  Systems  and  Training  Systems  for  all  U.S. 
Navy  Surface  Combatants,  including  Aircraft  Carriers,  Cruisers,  Destroyers,  Frigates,  Amphibious 
Ships,  and  auxiliaries.  In  July,  2006,  Mr.  Pitts  returned  to  NSWCDD  to  form  and  head  the  Warfare 
Systems  Department.  While  in  this  position,  he  maintained  his  personal  technical  involvement  as  the 
certification  official  for  Surface  Navy  Combat  Systems.  He  also  served  as  chair  of  the  Combat  System 
Configuration  Control  Board  and  chair  of  the  Mission  Readiness  Review  for  Operation  Burnt  Frost, 
the  killing  of  inoperative  satellite  USA  193. 

Mr.  Pitts  has  been  a  guest  speaker/lecturer/symposium  panelist  at  many  NAVSEA-level  and  DoD 
symposiums,  conferences  and  at  the  Naval  Postgraduate  School,  the  Defense  Systems  Management 
College,  and  the  National  Defense  University.  For  19  years  Mr.  Pitts  was  the  sole  certification 
authority  of  all  Aegis  Combat  System  computer  programs  for  fleet  use.  He  retired  from  the  U.S.  Civil 
Service  in  September  2008,  with  over  40  years  of  service  to  the  Navy. 
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Comparing  Software  Acquisition  Models  Against  Each 
Other:  The  "Build"  vs.  "Buy"  vs.  "Rent"  Trade  Study 


Ron  Kohl — Kohl  has  been  involved  in  the  large  systems  integration  business  for  over  30  years.  Kohl 
started  by  working  on  NASA’s  Space  Shuttle  and  Space  Station  programs  (Onboard  Flight  Software 
systems)  while  with  IBM’s  Federal  Systems  Division  in  Houston.  He  has  spent  several  years  with 
Loral’s  and  Lockheed  Martin’s  Federal  Systems  Headquarters  technical  staff,  and  then  with  Lockheed 
Martin’s  Software  and  Systems  Resource  Center  (SSRC).  Kohl  worked  with  Titan  Systems  Co. — Civil 
Government  Services  Group  as  the  chief  systems  engineer  for  NASA  programs.  He  is  now  president 
of  R.  J  .Kohl  and  Associates,  a  consulting  firm  offering  systems  engineering  and  technical  project 
management  services  to  various  government  and  industry  customers.  Kohl’s  areas  of  space  interest 
include  LEO  infrastructure  services,  socio-cultural  aspects  of  space  settlements,  the  future  space- 
based  workforce,  and  the  commercialization  of  space.  He  is  a  member  of  the  AIAA  Space 
Colonization  Technical  Committee.  Kohl’s  technical  interests  include  systems  engineering,  software 
engineering,  risk  management,  COTS-based  systems,  and  measurements.  He  is  a  member  of  the 
IEEE  Reliability  Society,  the  IEEE  Standards  Association,  and  the  AIAA  Computer  Systems  Technical 
Committee.  Kohl  has  a  BS  in  mathematics  from  the  University  of  Wisconsin-Oshkosh  and  an  MS  in 
mathematics  from  Southern  Illinois  University-Edwardsville.  [rjkohl@prodigy.net] 

Abstract 

Software  can  currently  be  acquired  in  three  different  methods.  The  first  is  to  have  software 
custom  built/developed  to  match  a  particular  specification/requirement.  We  shall  refer  to  this 
option  as  “make.”  The  second  is  to  purchase  a  software  product  from  a  vendor/supplier.  We 
shall  refer  to  this  option  as  “buy.”  The  third  is  to  rent/outsource  the  use  of  a  software  product 
or  a  software  development  environment  from  a  third-party  supplier.  We  shall  refer  to  this 
option  as  “rent.”  It  seems  that  what  is  lacking  is  some  guidance  to  help  acquirers  decide 
which  of  these  three  software  acquisition  approaches  to  consider  and,  eventually,  to  select. 
This  lack  of  objective,  quantitative  guidance  (including  risks  associated  with  each  option, 
decisions  needed  for  each  option,  etc.)  causes  acquirers  to  sometimes  make  ill-informed 
decisions  about  which  acquisition  method  to  use.  This  paper  identifies  some  of  the 
differences  between  these  three  acquisition  models  as  mapped  against  several  life  cycle 
phases  and  project  activities,  and  then  identifies  risks  associated  with  the  “rent”  option. 

Overview 

Software  can  currently  be  acquired  in  three  reasonably  different  methods  and  each 
of  these  methods  has  different  benefits,  risks,  and  decisions  to  address  as  part  of  a  software 
acquisition  activity. 

The  first  is  to  have  software  custom  built/developed  to  match  a  particular 
specification/requirement.  We  shall  refer  to  this  option  as  “build.” 

The  second  is  to  purchase  a  software  product  from  a  vendor/supplier.  Commercial 
off-the-shelf  (COTS)  software  is  one  such  example.  There  are  other  types  of  pre-built, 
packaged  software  products,  intended  to  be  used  in  mostly  “as-is”  condition  in  this  category. 
We  shall  refer  to  this  option  as  “buy.” 

The  third  is  to  rent/outsource  the  use  of  a  software  product  or  a  software 
development  environment  from  a  third-party  supplier.  A  subset  of  the  Cloud  Computing  (CC) 
paradigm  would  be  considered  part  of  this  option.  Note  that  in  this  option  we  typically  refer 
to  the  software  acquired  as  a  “service,”  rather  than  a  product,  since  the  software  itself  is  not 
physically  obtained  by  the  acquiring  organization.  We  shall  refer  to  this  option  as  “rent.” 
While  some  definitions  of  CC  include  both  internal  and  external  CC  providers,  this  paper 
focuses  only  on  the  third-party  CC  model,  not  the  internal  CC  model. 
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While  IEEE  1062,  Recommended  Practice  for  Software  Acquisition,  provides  some 
comparison  between  the  “build”  and  “buy”  acquisition  options,  it  does  not  address  the  “rent” 
option,  which  is  the  focus  of  this  paper.  In  this  paper,  I  compare  certain  aspects  of  the  CC 
model  against  the  “build”  acquisition  model,  and  certain  other  aspects  of  the  CC  model 
against  the  “buy”  acquisition  model.  It  is  the  author’s  hope,  being  a  current  member  of  the 
IEEE  1062  update  team,  that  some  future  version  of  this  IEEE  standard  will  include 
information  about  this  CC  (i.e.,  “rent”)  option. 

So,  What’s  the  Problem? 

It  seems  that  what  is  lacking  or  not  fully  developed  is  some  guidance  to  help 
acquirers  decide  which  of  these  three  software  acquisition  approaches  to  consider  and 
eventually  select,  or  to  help  acquirers  assess  a  supplier’s  proposal  to  adopt  one  of  them. 

This  lack  of  objective,  quantitative  guidance  (including  risks  associated  with  each  option, 
decisions  needed  for  each  option,  etc.)  causes  acquirers  to  sometimes  make  ill-informed 
decisions  about  which  acquisition  method  to  use  (or  which  combination  of  methods  to  use). 
Creating  some  guidance  for  software  acquirers  that  will  help  them  make  effective  trade 
studies  leading  to  a  choice  of  one  (or  more)  of  these  acquisition  options  should  improve  the 
quality  of  both  the  software  to  be  acquired  and  the  overall  system  in  which  that  software  will 
be  executed.  This  paper  identifies  risks  with  the  three  major  categories  of  development,  test, 
and  maintenance,  and  suggests  some  differences  between  these  three  acquisition  models 
as  mapped  against  various  life  cycle  phases  and  project  activities. 

Definitions  for  CC  Options 

•  SaaS:  A  model  in  which  an  organization  basically  rents  an  application,  paying 
a  flat  monthly  fee  based  on  the  number  of  transactions,  users,  or  employees. 

•  laaS:  Off-loading  the  guts  of  an  organization’s  data  center,  such  as  servers 
and  networking,  to  a  cloud  provider.  It  is  attractive  to  organizations  that  do  not 
want  to  manage  their  infrastructure,  undertake  an  infrastructure  upgrade,  or 
deal  with  scalability  issues,  and  that  would  prefer  to  off-load  that  responsibility 
to  a  third  party. 

•  PaaS:  A  cloud  service  that  consists  of  an  entire  platform — user  interfaces, 
workflow  engines,  database  services,  and  security/authentication — complete 
with  tools  to  walk  you  through  the  application-building  process.  This  aspect  of 
CC  is  essentially  the  outsourcing  of  the  Software  Development  Environment 
(SDE)  for  a  custom-built  software  project. 

Example 

Here  is  an  example  of  a  software  need  and  how  these  three  acquisition  methods 
create  differences  to  consider.  Suppose  we  have  a  need  for  an  accounting  package  (for 
example,  General  Ledger  [GL]).  We  could  solicit  and  then  hire  a  software  developer  to 
create  a  custom-built  software  product  to  meet  our  needs.  Alternatively,  we  could  visit  the 
COTS  marketplace  and  try  out  a  few  different  GL  packages  and  then  select  one  to  meet  our 
needs.  Finally,  we  could  visit  the  Cloud  Computing  world  (and  other  “rent  a  software 
product”  options)  to  see  if  there  is  a  supplier  offering  the  use  of  a  GL  package  to  perform  our 
GL  processing.  We  could  assess  the  options  and  then  select  one  that  meets  our  needs. 

We  should  note  that  in  certain  application  domains,  we  may  not  have  all  of  these 
options  to  consider.  For  example,  if  we  need  a  unique  software  application  to  support  a  new 
spacecraft  system,  then  we  may  only  have  the  “build”  option.  Another  example  might  be  the 
need  for  an  integrated  accounting,  inventory  control,  and  logistics  software  product.  In  this 
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case,  we  may  have  only  the  “build”  and  “buy”  options,  as  there  may  not  yet  be  a  CC  supplier 
offering  such  an  integrated  package. 

So,  What  Should  We  Do  About  It? 

First,  I  present  a  set  of  risk  areas/questions  that  relate  to  software  acquisition  in 
general,  and,  in  some  cases,  to  the  CC  approach  in  particular.  This  list  provides  a  starting 
point  (but  is  not  intended  to  be  a  complete  list)  of  the  kinds  of  questions  that  acquirers 
should  be  asking  and,  depending  on  the  answers  received,  can  lead  to  identifying  risk  areas 
associated  with  a  particular  acquisition  approach.  I  then  map  certain  of  these  risks/questions 
to  three  major  phases  in  software  acquisition  (pre-solution  selection  phase,  solution 
selection  and  system  development  phase,  operations  (Ops)/maintenance  phase)  and 
provide  more  detail. 

Risk  Areas 

1 .  Access  to  assets/resources — What  if  my  access  to  these  services/products  is 
interrupted  or  lost? 

2.  Information  security — Is  my  property  appropriately  protected? 

3.  Support  to  users — What  if  the  acquirer/users  claim  they  found  a  problem  in 
the  service/product?  If  the  acquirer  would  like  to  request  a  new 
feature/capability,  how  receptive  is  the  vendor/supplier  to  such  requests? 

4.  CC  service  “users  groups” — Many  COTS  vendors  have  “user  groups”  that 
allow  the  users  of  the  product  to  gather  together  to  share  information.  Is  there 
anything  equivalent  in  the  CC  world? 

5.  Managing  version/upgrade  control — What  happens  when  the  CC  vendor 
decides  to  make  changes  to  the  services  or  products  that  they  rent  out  to 
acquirers?  How  many  such  versions  are  supported  by  the  vendor/supplier? 
What  happens  if  an  acquirer  is  using  a  version  that  is  no  longer  supported? 

6.  Vendor  viability — What  if  my  CC  vendor  goes  out  of  business? 

7.  Performance — Am  I  getting  the  responsiveness  and  availability  that  was 
advertised? 

8.  Capacity — Am  I  getting  the  advertised  number  of  users  using  the  CC 
services?  Do  the  CC  services  provide  enough  space/resources  to  support  my 
capacity  needs? 

9.  Role  of  internal  IT  staff/services  in  a  CC  world 

a.  Internal  staff  may  have  to  act  as  a  liaison  to  the  CC  vendor/services 

b.  Internal  staff  may  have  to  learn  new  skills  (technical,  business, 
customer  support,  etc.) 

10.  Consistency  of  internal  IT  assets/resources/staff  versus  CC  vendor  services 

a.  Do  internal  staff  skills  need  to  be  expanded? 

b.  Does  new  staff  need  to  be  hired? 

c.  Are  internal  tools/methods  consistent  with  CC  vendor 
tools/methodologies  (e.g.,  is  my  CM  tool  compatible  with  the  CC 
vendor’s  CM  tool)? 

Three  Major  Phases  in  a  Software  Acquisition  Life  Cycle 

For  the  acquisition  of  any  software  product  or  software  service,  there  are  a  common 
set  of  interests,  such  as  assessing  the  quality  of  the  vendor,  the  likely  quality  of  the  product, 
security  and  availability  issues,  and  so  forth.  But  from  a  software  acquirer’s  perspective,  the 
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nature  of  the  risks  and  any  possible  mitigation  steps  may  differ,  depending  on  where  in  the 
overall  software  acquisition  and  project  life  cycle  the  acquirer  is  at.  I  identify  these  areas  of 
risk  based  on  the  three  phases  discussed  in  the  following  list,  and  pose  some  questions  that 
can  help  to  determine  whether  a  given  risk  area  is  applicable  to  or  is  likely  to  emerge  in  a 
given  phase. 

1 .  Prior  to  selecting  a  product/service  and  the  appropriate  supplier 

a.  Quality  of  the  software  product:  Has  this  provider  done  similar  work  on 
past  projects?  Was  that  work  considered  acceptable  by  the  customer? 
Is  there  a  large  user  base? 

b.  Technical  quality  of  the  supplier  of  the  software  product:  What  is  the 
quality  of  the  technical  processes  used  by  this  supplier  (e.g.,  CMMI 
Level  3,  ISO  9001 )?  Has  this  supplier  done  other  projects  in  this 
application  domain?  If  yes,  will  the  supplier  share  details  about  those 
other  projects?  How  does  the  supplier  collect  defect  reports  from  the 
user  base?  Are  those  defect  reports  shared  with  the  user  base?  Will 
the  supplier  share  the  criteria  by  which  they  declare  an  alleged  defect 
to  be  an  actual  defect  or  by  which  it  gets  rejected? 

c.  Business  viability  of  the  supplier  of  the  software  product:  Has  this 
provider  been  in  business  for  a  long  time  or  not?  How  big  a  portion  of 
the  marketplace  does  this  provider  “own”?  Does  this  provider  have  a 
user  base  that  has  its  own  user’s  group?  What  is  the  likelihood  of  this 
provider  being  around  long  enough  to  provide  support  during  the 
development  and  Ops/maintenance  phases? 

d.  Security:  Is  any  information/data  that  is  produced  by  this  product  or 
service  fully  protected?  Is  there  any  concern  about  malware,  hacker 
access,  or  other  intrusion  threats? 

e.  Availability:  Can  I  always  get  to  my  information?  Can  I  always  have 
use  of  the  product/service?  Can  the  product/service  run  continuously 
for  an  advertised  length  of  time  before  failing  (i.e. ,  is  there  an 
availability  or  reliability  attribute)? 

f.  Supportability:  Does  the  provider  offer  product  support/training?  If 
there  are  multiple  versions/releases  of  the  product,  which  ones  will  be 
supported  and  for  how  long?  What  are  my  options  if  I  am  using  a 
version/release  that  no  longer  is  supported  by  the  provider? 

g.  What  are  my  product/service  acceptability  criteria?  Some  of  the 
above?  Other  criteria? 

h.  What  contracting  vehicles  are  available  to  me  to  acquire  this  software, 
or  access  use  of  this  software,  from  this  supplier? 

2.  After  selecting  product/service/supplier  combination,  but  prior  to  deployment 
into  Operations;  This  includes  any  system  development/test/integration 
processes  that  integrate  this  product/service  into  a  larger  system. 

a.  How  well  does  the  product/service  working  after  I  have  acquired  it? 

b.  What  measures  should  I  implement  to  monitor  the  use  of  this 
product/service? 

c.  What  if  I  discover  a  defect?  What  if  the  supplier  does  not  agree  with 
me  about  an  alleged  defect? 

d.  How  responsive  is  the  supplier  to  my  questions  or  other  issues? 
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3.  After  system  deployment  into  the  Ops/maintenance  mode 

a.  What  if  I  discover  a  defect?  What  if  the  supplier  does  not  agree  with 
me  about  an  alleged  defect? 

b.  What  if  the  supplier  creates  a  new  version/release  of  the  product?  Do 
I  need  to  upgrade  or  can  I  remain  on  the  current  version?  How  many 
versions  will  the  supplier  support?  What  are  my  options  if  a  version 
that  I  am  using  is  no  longer  supported  by  the  supplier? 

c.  What  are  the  user’s  satisfaction  criteria?  Are  these  important  to  the 
vendor/supplier/developer? 

Comparing  Different  Acquisition  Methods 

The  three  different  acquisition  methods  (make,  buy,  rent)  have  different  risk  areas 
and  different  decisions  that  must  be  made  when  these  methods  are  being  considered  for 
adoption  and,  eventually,  selected  as  a  preferred  method.  While  these  three  methods  have 
many  differences,  it  is  possible,  even  likely,  that  a  given  project/system  will  use  more  than 
one  such  approach.  In  fact,  the  overall  acquisition  approach  could  easily  use  more  than  one 
of  these  approaches  and,  thus,  acquirers  will  likely  need  to  consider  the  risks/decisions  for 
each  approach,  as  well  as  the  combination  of  two  or  more. 

laaS  (Infrastructure  as  a  Service)  Versus  Internal  IT  Services 

I  will  defer  this  comparison  to  a  future  paper,  due  to  the  lack  of  direct  insight  into  this 
aspect  of  CC. 

SaaS  (Software  as  a  Service)  Versus  COTS 

This  comparison  will  look  at  how  an  organization  can  compare  “buy  a  commercial 
application”  versus  “rent  access  to  and  use  of  a  commercial  application.”  In  this  comparison, 
I  assume  that  the  application  is  essentially  the  same  application,  and  I  focus  on  the 
differences  in  these  two  acquisition  models. 

The  major  differences  between  these  two  approaches  is  that  in  the  COTS-based 
approach,  the  acquirer  takes  possession  of  the  software  product  and,  thus,  gains  greater 
control  over  access  to  it  and  utilization  of  it.  But  this  control  factor  is  balanced  against  the 
cost  of  acquiring  the  product,  ensuring  that  there  is  internal  staff  (and  skills)  that  can  provide 
necessary  support  to  the  product.  In  addition,  product  version  control  (updates)  is  less 
impacted  in  the  COTS-based  approach  due  to  having  possession  of  a  given  version  of  the 
product.  If  the  COTS  vendor  decides  to  create  new  versions  or  update/upgrade  existing  and 
deployed  versions,  the  acquirer  can  choose  to  accept  the  change  or  defer  that  change. 

Such  options  may  not  be  available  if  the  CC  vendor  chooses  to  only  offer  the  updated 
version  or  a  newer  version  of  their  product. 

PaaS  (Platform  as  a  Service)  Versus  Custom-Built  Solution 

This  comparison  looks  at  the  differences  between  custom  building  an  application 
using  internal  resources  and  internal  staff  versus  using  a  rented  development  environment 
(e.g.,  tools,  skills,  resources)  with  both  internal  staff  and  some  staff  from  the  PaaS  vendor. 

The  major  difference  here  is,  again,  a  matter  of  control  of  the  development 
environment  and  the  supporting  tools  and  skills.  In  the  custom-built  software  approach,  the 
acquirer  has  possession  of  the  tools,  the  methodologies,  and  the  staff/skills,  and  can  control 
all  of  these  aspects  of  this  acquisition  approach.  In  the  PaaS  approach,  the  acquirer  trades 
off  this  control  for  a,  hopefully,  reduced  total  cost  of  “development”  and  a  shorter  period  to 
create,  and  start  to  use,  the  development  environment. 
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Open  Questions  and  Next  Steps 

Open  Questions 

•  What  role  should  the  government  (e.g.,  NIST,  etc.)  take  in  further  expanding 
and  enhancing  these  risk  areas  and  possible  risk-mitigation  approaches?  Are 
there  other  “communities  of  practice”  that  can  contribute  to  this  objective, 
honest-broker  perspective  on  CC? 

•  Can  these  comparisons  be  expanded  across  all  three  methods  to  provide  a 
“big  picture,”  side-by-side  view  of  these  three  software  acquisition 
approaches? 

Next  Steps 

•  Expanding  the  risk  lists  with  additional  risk  areas  (under  construction) 

•  Developing  a  set  of  candidate  risk-mitigation  approaches  to  deal  with  these 
various  risks  (under  construction) 

•  Investigating  the  ability  to  compare  laaS  against  other  software  or  systems 
acquisition  approaches 

References/Resources 

1.  The  National  Institute  of  Standards  and  Technology  (NIST; 
http://www.nist.gov/itl/cloud/)  is  a  leader  in  identifying  the  government’s 
interests  in  Cloud  Computing.  Their  website  describes  the  various  activities 
that  they  are  leading  or  supporting  related  to  furthering  the  successful  use  of 
CC. 

2.  The  Cloud  Standards  Customer  Council  (http://www.doud- 
council.org/index.htm)  is  an  industry  consortium  intended  to  improve  the 
adoption  and  use  of  CC. 
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What  is  the  problem? 


Cloud  Computing  (CC)  has  expanded  the  'make 
vs  buy'  trade  study  to  'make  vs  buy  vs  rent' 

Yet  there  is  little  guidance  to  help  SW  acquirers 
make  this  trade  study 

—  What  are  the  risks  associated  with  CC? 

-  How  can  I  compare  CC  against  the  other  options? 


What  is  the  solution? 


Understand  the  risks  associated  with  CC 

Understand  how  these  CC  risks  relate  to 
different  phases  of  SW  acquisition 

Be  able  to  compare  riskyness  of  each 
acquisition  approach  against  the  others 


Definitions 


Cloud  Computing  (CC)  -  computing/SW 
products  or  services  for  'rent'  from  some 
vendor 

-  SaaS  (Software  as  a  Service):  renting  the  use  of  a 
s/w  application  (e.g.  accounting) 

-  PaaS  (Platform  as  a  Service):  renting  the  use  of  s/w 
tools  to  build  your  own  s/w 

-  laaS  (Infrastructure  as  a  Service):  use  of  physical 
computing  products/services  to  support  an  org  (e.g. 
IT  services,  data  servers,  etc) 


Risks  associated  with  CC 


Access  to  resources  -  does  the  vendor  provide 
'guaranteed'  access  or  alternative  access? 

Resource  updates  -  how  does  the  CC  vendor 
manage  updates  to  their  products/services? 

Info  Security  -  how  does  the  CC  vendor  plan  to 
protect  your  information? 

Vendor  viability- what  if  the  vendor  goes  out  of 
business?  What  are  your  options? 

There  are  more  in  the  paper  and  more  that  may 
not  yet  be  identified  or  encountered. 


how  these  CC  risks  relate  to  different 
phases  of  SW  acquisition 

SW  acquisition  phases: 

-  Prior  to  acquiring  a  SW  product/service 

-  After  acquiring  SW  product/service  but  before 
operations  (e.g.  development,  test,  integration, 
etc) 

-  After  SW  product/service  is  put  into  operations 

See  'IEEE  P1062  Recommended  Practice  for 
Software  Acquisition'  for  more  details 

This  list  is  'under  construction'  and  not  part  of  this 
paper 


Compare  riskiness  of  each  acquisition 
approach  against  the  others  (1) 

SaaS  vs  COTS  (aka  Tent  access  to  vs  buy'  a 
commercial  SW  product): 

-  possession  of  product  vs  access  to  product 

-  Internal  staff/skill  needs 

-  Update/version  control  options 

-  Information  protection 

-  Managing  SW  product/service  defects 


Compare  riskiness  of  each  acquisition 
approach  against  the  others  (2) 

PaaS  vs  Custom  Built  SW  (aka  'rent  access  to 
SW  dev  environment  vs  internal  SW  dev 
environment'): 

-  possession  of  product  vs  access  to  product 

-  Capacity  of  users  to  access  tools 

-  Update/version  control  options 

-  Information  protection 

-  Tool  defects  management 


Next  Steps 


Improve  list  of  CC  risk  areas,  map  to  phases 

Develop/improve  risk  mitigations  for  these  CC 
risks 

Improve  comparisons  between  'make  vs  buy 
vs  rent'  acquisition  models 

—  Incorporate  into  IEEE  1062  Std? 

Government  role  in  addressing  these  issues 
(e.g.  NIST,  etc) 


